Eye gaze for spoken language understanding in multi-modal conversational interactions.” ICMI

When humans converse with each other, they naturally amal-gamate information from multiple modalities (i.e., speech, gestures, speech prosody, facial expressions, and eye gaze). This paper focuses on eye gaze and its combination with speech. We develop a model that resolves references to vi-sual (screen) elements in a conversational web browsing sys-tem. The system detects eye gaze, recognizes speech, and then interprets the user’s browsing intent (e.g., click on a specific element) through a combination of spoken language understanding and eye gaze tracking. We experiment with multi-turn interactions collected in a wizard-of-Oz scenario where users are asked to perform several web-browsing tasks. We compare several gaze features and evaluate their effec-tiveness when combined with speech-based lexical features. The resulting multi-modal system not only increases user in-tent (turn) accuracy by 17%, but also resolves the referring expression ambiguity commonly observed in dialog systems with a 10 % increase in F-measure